Systems and methods for displaying three-dimensional images

ABSTRACT

Systems and methods for displaying three-dimensional (3D) images are described. In particular, the systems can include a display block made from a transparent material with optical elements three-dimensionally disposed therein. Each optical element becomes luminous when illuminated by a light ray. The systems can also include a computing device configured to generate two-dimensional (2D) images formatted to create 3D images when projected on the display block, by a video projector coupled to the computing device. The video projector is configured to project the 2D images on the block to create the 3D images by causing a set of the passive optical elements to become luminous. Various other systems and methods are described for displaying 3D images.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/569,684, filed on Sep. 29, 2009, which is a continuation of U.S.patent application Ser. No. 11/666,228, filed on Apr. 7, 2008, which isthe U.S. National Phase application of International Application No.PCT/US2005/038321, filed on Oct. 25, 2005, which claims the benefit ofU.S. Provisional Patent application No. 60/621,837, filed on Oct. 25,2004, U.S. Provisional Patent application No. 60/651,740, filed on Feb.10, 2005, and U.S. Provisional Patent application No. 60/726,301, filedon Oct. 12, 2005, which are hereby incorporated by reference herein intheir entireties.

FIELD OF THE INVENTION

The present invention relates to systems and methods for displayingthree-dimensional (3D) images. More particularly, the present inventionrelates to systems and methods for displaying 3D images using passiveoptical elements in a display block. The present invention also relatesto systems and methods for displaying 3D images using anamorphic lightengines and/or reflective elements having curved surfaces.

BACKGROUND OF THE INVENTION

Systems for displaying images (for both static and video images) havebecome a ubiquitous part of our everyday lives. For example, televisionsprovide viewers with news and entertainment. Display monitors used incomputers and cellular phones enable users to interact with a variety ofdevices and various forms of information via images displayed on themonitors. High quality digital display systems have also emerged aspossible replacements for physical media such as photographs andpaintings. Recently, home theater systems with large projection displaysystems allow viewers to enjoy theater-like experience in their homes.

Despite the advancement of technologies relating to display systems,most conventional display systems only display two-dimensional (2D)images. A viewer, however, perceives the world in three dimensions byperceiving depth in addition to the horizontal and vertical dimensions.Because 2D images do not contain depth information, they appear to beless realistic to a viewer. A system that can display static or dynamic3D images in high resolution is therefore desirable over 2D displaysystems. Moreover, in some situations, it is also desirable that a 3Ddisplay system simultaneously provide different perspectives of a 3Dscene to viewers who are located at different angles with respect to the3D scene. Such a system also allows a viewer to move around the 3D sceneand gain different perspectives of the scene.

Several approaches have been used or proposed to display 3D images. Oneapproach is to project two different images on one screen. The twodifferent images contain the same scene captured from two differentangles. A viewer is required to wears glasses that separate the combinedimage into the two different images. In particular, the glasses causeeach eye of the viewer to perceive one of the two different images. Thisseparation is possible because each image uses a distinct color (e.g.,red and blue) or polarization. However, this approach suffers from thedrawback that viewers must wear glasses in order to have a 3Dexperience.

A second conventional approach is to combine multiple 2D images of ascene, captured from different angles, into a single 2D image. In thisapproach, a set of adjacent pixels in the combined 2D image plays therole of a single pixel. Each pixel in the set of pixels corresponds tothe same point in a scene, but has different brightness and colorcorresponding to a different perspective. A pinhole or a slit is placedat some distance from the set of pixels. For each point in the scene tobe displayed, the pinhole or slit passes different color and brightnessin different angles. Therefore, the eyes of a viewer perceive imagesthat correspond to two different perspectives of a 3D scene. As a viewermoves around, the viewer also obtains different perspectives of the 3Dscene being displayed. Instead of a pinhole or a slit, sometimes a lensis used. However, this approach suffers from the drawback that spatialresolution is significantly reduced.

A third conventional approach is to trade-off brightness resolution forgenerating the needed directional variation in displayed colors orbrightness. In this approach, a screen is rotated at a very high speed.The rotating screen covers a 3D region of space. Each point in the 3Dregion is illuminated only when the screen passes through that point.The screen completes at least one full rotation during the time the eyeintegrates a single image. Therefore, the two eyes perceive images thatcorrespond to two different perspectives of a 3D scene. In this case, anenormous amount of light energy is needed for the scene to appear crispand bright. In addition, it requires the continuous mechanical movementof an entire projection system. As a result, it is difficult to scalesuch an approach to cover reasonably large display spaces. Finally, thisapproach is limited because it does not adequately handle points thatare hidden from the viewer. Because one does not know a priori where theviewer is located, all points in the 3D scene are lit. Hence, the pointsthat should be hidden can be seen “through” other visible points.

Yet another conventional approach is to use a display block formed by aset of liquid crystal sheets stacked together. The cells of the sheetsare of the “scattering” type. A high frame rate projector is used toilluminate the stack of sheets where each projected frame is scatteredby a single liquid crystal sheet while the remaining sheets are fullytransparent and hence let light pass through to the viewer. Because theintegration time of the eye is greater than the time it takes toilluminate all the sheets, the viewer perceives a volume that is lit upat the appropriate locations. This approach also suffers from thedrawback that an enormous amount of light energy is required to create abright 3D scene. In addition, points that should be hidden are alwaysvisible to the viewer.

In other conventional volumetric display systems, the display blocks aremade of materials that can locally respond to specific types ofillumination. In one example, fluorescent materials are used that glowwhen illuminated with laser beams shone from multiple directions. Suchdisplays do not create a four-dimensional light field because thedirectional radiance of each point cannot be controlled. As a result,the displayed image is a collection of translucent (ghost-like) glowingpoints of light that are visible from all directions. Finally,holographic methods have been suggested several times in the past as apossible alternative. Unfortunately, conventional holographic displayscan only display low quality images.

SUMMARY OF THE INVENTION

Various embodiments of the present invention provide 3D display systemsand methods that overcome various shortcomings discussed above. Forinstance, the 3D display systems of various of embodiments of thepresent invention do not require the use of specialized glasses or alarge amount of light power. Moreover, the 3D display systems of variousof embodiments of the present invention can provide differentperspectives of a displayed 3D scene to viewers located at differentviewing angles and/or not show points that should be hidden fromviewers.

These and other advantages of various embodiments of the presentinvention are provided by using, among other things, a display devicethat includes a display block made from a transparent material andpassive optical elements. The passive optical elements arethree-dimensionally disposed within the display block. Each opticalelement becomes luminous when illuminated by a light ray. The displayblock can be a solid three-dimensional block with passive opticalelements that are localized damages produced by an etching process.Alternatively, the display block can be stacked layers with the passiveoptical elements embedded between the stacked layers. Here, the passiveoptical elements can be Lambertian reflectors. The passive opticalelements can form a 3D grid with regular or irregular patterns.Alternatively, the passive optical elements can form a 3D object. Anexample of the 3D object is a mask of a human face. Various embodimentsof the present invention may also include a computing device configuredto generate 2D images formatted to create 3D images when projected onthe display block, and a video projector coupled to the computingdevice. The projector is configured to project the 2D images on theblock to create the 3D images by causing a set of the passive opticalelements to become luminous.

Embodiments of the present invention also provide various methods todisplay 3D images. An example method may include the step of generating2D images formatted to create 3D images when projected on a displayblock made from transparent material with three-dimensionally disposedpassive optical elements. Here, each passive optical element becomesluminous when illuminated by a light ray. The example method may alsoinclude the step of projecting the 2D images on the block to create the3D images by causing a set of the passive optical elements to becomeluminous.

Various embodiments of the present invention provide another system fordisplaying 3D images. The system includes an array of reflectivespheres, a camera having a focal point, and a projector having aprojection center. The system also includes a beam splitter interposedbetween the camera and the projector such that the focal point of thecamera is co-located with the projection center of the projector withrespect to the array of reflective spheres.

Various embodiments of the present invention further provide yet anothersystem for displaying 3D images. The system includes an array of videoprojectors. Each of the video projectors includes an anarnorphic lensconfigured to compress images in one direction. The system also includesa diffusive screen to project images from the array of video projectors,and a set of lenticular lenses disposed near the diffusive screen, tothereby create 3D images from the compressed images when perceived by aviewer.

BRIEF DESCRIPTION OF THE DRAWINGS

The Detailed Description including the description of variousembodiments of the invention will be best understood when read inreference to the accompanying figures wherein:

FIG. 1 is a perspective view illustrating a volumetric display systemusing passive optical elements according to various embodiments of thepresent invention;

FIG. 2 is a cross-sectional view of a volumetric display systemillustrated in FIG. 1;

FIG. 3 a cross-sectional view of another volumetric display systemaccording to various embodiments of the present invention;

FIG. 4 is a diagram illustrating an example of a volumetric displaysystem according to various embodiments of the present invention;

FIG. 5 is a diagram illustrating a relationship between passive opticalelements and projector pixels in an example of a volumetric displaysystem according to various embodiments of the present invention;

FIG. 6 is a flow chart illustrating the steps for displaying 3D imagesusing a volumetric display system according to various embodiments ofthe present invention;

FIG. 7 is a cross-sectional view of an example of a volumetric displaysystem, wherein the grid of passive optical elements forms a regularpattern according to various embodiments of the present invention;

FIG. 8 is a cross-sectional view of a volumetric display systemgenerally illustrated in FIG. 7, wherein the locations of the passiveoptical elements are perturbed according to various embodiments of thepresent invention;

FIG. 9 is a cross-sectional view of a volumetric display systemgenerally illustrated in FIG. 7, wherein the locations of the passiveoptical elements are perturbed according to various embodiments of thepresent invention;

FIG. 10 is a cross-sectional view of a volumetric display systemgenerally illustrated in FIG. 7, wherein the locations of the passiveoptical elements are perturbed according to various embodiments of thepresent invention;

FIG. 11 is a perspective view of an example of a volumetric displaysystem according to various embodiments of the present invention;

FIG. 12 is a perspective view of a volumetric display system displayinga 3D image according to various embodiments of the present invention;

FIG. 13 is a cross-sectional view of an example of a volumetric displaysystem according to various embodiments of the present invention;

FIG. 14 is a cross-sectional view of an example of a volumetric displaysystem according to various embodiments of the present invention;

FIG. 15 is a system diagram of an interactive volumetric display systemaccording to various embodiments of the present invention;

FIG. 16 is a perspective view of a display block used in a 3D gamesystem according to various embodiments of the present invention;

FIG. 17 is a diagram further illustrating the display block in FIG. 16;

FIG. 18 is a perspective view of a volumetric display system using alaser pointer according to various embodiments of the present invention;

FIG. 19 is a diagram illustrating passive optical elements that areLambertian reflectors, located on a transparent layer according tovarious embodiments of the present invention;

FIG. 20 is a diagram illustrating passive optical elements that arespecular hemispherical reflectors, located on a transparent layeraccording to various embodiments of the present invention;

FIG. 21 is a diagram illustrating passive optical elements that arelight diffusers, located on a transparent layer according to variousembodiments of the present invention;

FIG. 22 is a diagram of a volumetric display system using reflective:optical elements according to various embodiments of the presentinvention;

FIG. 23 is a diagram of a volumetric display system using diffusiveoptical elements according to various embodiments of the presentinvention;

FIG. 24 is a diagram of a compact volumetric display system usingdiffusive optical elements according to various embodiments of thepresent invention;

FIG. 25 is a diagram of a volumetric display system using multiple lightsources according to various embodiments of the present invention;

FIG. 26 is a diagram of a volumetric display system for animating 3Dimages of a face according to various embodiments of the presentinvention;

FIG. 27 is a flow chart illustrating the process of animating a 3D faceimage using a volumetric display system according to various embodimentsof the present invention;

FIG. 28 is a cross-sectional view of a volumetric display system,wherein passive optical elements form a sphere according to variousembodiments of the present invention;

FIG. 29 is a diagram illustrating an example of projecting two images ina volumetric display system illustrated in FIG. 28 according to variousembodiments of the present invention;

FIG. 30 is a diagram illustrating the operation of a 3D display systemusing anamorphic light engines according to various embodiments of thepresent invention;

FIG. 31 is a flow chart illustrating the operation of a 3D displaysystem using anamorphic light engines according to various embodimentsof the present invention;

FIG. 32 is a perspective view of a single anamorphic light engine in thesystem illustrated in FIG. 30;

FIG. 33 is a diagram illustrating the compression of an image by ananamorphic light engine according to various embodiments of the presentinvention;

FIG. 34 a perspective view of an anamorphic light engine using a curvedmirror according to various embodiments of the present invention;

FIG. 35 is a perspective view of an array of anamorphic light engines inoperation according to various embodiments of the present invention;

FIG. 36 is a perspective view of a portion of a 3D display system usinganamorphic light engines according to various embodiments of the presentinvention;

FIG. 37 is a perspective view of a portion of a 3D display system usinganamorphic light engines according to various embodiments of the presentinvention;

FIG. 38 is a perspective view of a compact 3D display system usinganamorphic light engines according to various embodiments of the presentinvention;

FIG. 39 is a diagram of a compact anamorphic light engine according tovarious embodiments of the present invention;

FIG. 40 is a perspective view of a 3D display system using light fieldelements according to various embodiments of the present invention;

FIG. 41 is a diagram illustrating the operation of a 3D display systemusing light field elements according to various embodiments of thepresent invention;

FIG. 42 is a diagram illustrating a representation of a 4D light fieldaccording to various embodiments of the present invention;

FIG. 43 is a perspective view of a 3D display system using light fieldelements according to various embodiments of the present invention;

FIG. 44 is a cross-sectional view of a compact bi-directional lightfield element according to various embodiments of the present invention;

FIG. 45 is a perspective view of a 3D display system using a 2D array ofbi-directional light field elements illustrated in FIG. 44 according tovarious embodiments of the present invention;

FIG. 46 is a perspective view of a 3D display system usingunidirectional light field elements according to various embodiments ofthe present invention;

FIG. 47 is a perspective view of another 3D display system using lightfield elements according to various embodiments of the presentinvention; and

FIG. 48 is a flow chart illustrating the steps for display 3D scenesusing a system illustrated in FIG. 40 according to various embodimentsof the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Volumetric Display Systems Using Passive Optical Elements

According to various embodiments of the present invention, a system fordisplaying 3D images can include a display block containing passiveoptical elements. In some embodiments, a display block may be formedfrom a solid block of transparent glass or plastic. Within this block,optical elements may be etched using a well-known technique calledlaser-induced damage (LID), where the elements are etched within thematerial by using a pulsed laser beam that is periodically focused atthe locations of the elements. This technique of etching is well knownin the art and is typically used to create static 3D images within glassand plastic blocks. Each etched element scatters light (and hence glows)when a ray of light illuminates it. Alternatively, a display block maybe formed by stacking sheets of a transparent material together withoptical elements embedded between the sheets. These sheets may be spacedusing any suitable technique.

FIG. 1 illustrates an example of a volumetric display system usingpassive optical elements. As shown, the system includes a display block101 (which may be a glass block, for example from Lazart Inc.), a lens103 for creating parallel illumination rays (for example, a Fresnellens, Model #37, manufactured by Fresnel Technologies, Inc.), a flatredirection minor 105 for folding the light rays to enable smallerpackaging, and a video projector 107 (for example, Model U5-232,manufactured by Plus Vision Corporation). The display block 101 containsa group of optical elements 109 (for example, etched in a glass blockusing LID).

In operation, the redirection minor 105 redirects the image produced bythe video projector 107 in the direction of the Fresnel lens 103. TheFresnel lens 103 then converts the diverging set of rays 111 from thevideo projector 107 into a parallel set of rays 113. These parallel rays113 are focused close to or within the display block 101, causing theelements 109 within the display block 101 to become luminous.

As generally illustrated, the elements 109 are arranged in a 3D grid ofpoints as described below. Although the term “grid” is used herein, itshould be apparent to one of ordinary skill in the art the elements maybe positioned at any suitable locations within the grid as describedherein and need not be located with any regular spacing.

Because a video projector 107 may be used, the illumination of elements109 generated by the video projector 107 can be changed at the framerate of the projector 107. Therefore, this system can be used to producedynamic, color 3D images within the display block 101. For example,still images and moving video may be created in the display block 101.In some embodiments, the video projector 107 may be controlled using asuitable video signal source such as a computer, a DVD player, etc.

FIG. 2 is a cross-sectional view illustrating an example of a possibleoptical layout of the system illustrated in FIG. 1. In this layout, theFresnel lens 103 can be tilted 14.85 degrees relative to the opticalaxis of the projector 107. The redirection mirror 105 can be tilted37.575 degrees relative to the vertical axis of the figure. It may benecessary to adjust the position of the redirection mirror 105 duringassembly of the system by rotating the redirection mirror around thehorizontal axis of the figure that is perpendicular to the figure'splane. The clear aperture of the redirection mirror 105 may havedimensions of 200 mm x 200 mm, which is the same as the horizontal crosssection of the display block 101, as well as the dimension of theFresnel lens 103. Several millimeters can be added to the mirrordimensions for mirror mounting. Therefore, the overall size of theredirection mirror can be 210 mm ×210 mm. Projector 107 can have anobjective focus of 18.4 millimeters. The relative position of variouscomponents of the system is also shown in this figure.

FIG. 3 illustrates another example of a possible layout of a volumetricdisplay system. Unlike the volumetric display system illustrated inFIGS. 1 and 2, this system does not include a redirection mirror. Thevideo projector 301, the Fresnel lens 303, and the display block 305 canbe of the same models as in FIG. 1. The Fresnel lens 303 may havedimensions 267×267 millimeters and a focus of 360 millimeters. Projector301 can have an objective focus of 18.4 millimeters. The relativeposition of various components of the system is also shown in thisfigure.

Systems With Regularly Spaced Passive Optical Elements

FIG. 4 is a diagram illustrating an example of a volumetric displaysystem. In this example, passive optical elements lie on parallellayers. The figure shows three layers 401 a, 401 b, and 401 c of opticalelements as a portion of a display block. Also included in the figure isa portion of a light engine, represented by a group of light enginepixels 407 illuminating the corresponding optical elements. A lightengine, as used in this context, is an apparatus that produces lightrays for displaying images. A light engine pixel, as used in thiscontext, is a unit element in a light engine that is capable ofproducing a distinct light ray (i.e., a light ray of a unique color andbrightness independent of other light rays). Within each layer, theoptical elements are equally spaced so that they form a regular pattern.As shown, the layers 401 a, 401 b, and 401 c may be positioned withequal distance between neighboring layers. The patterns of opticalelements in the different layers are offset with respect to each otherso that parallel light rays from projector light source 407 can bedirected at the optical elements without interference from the opticalelements of the other layers. For example, parallel light rays 403 a,403 b and 403 c are directed at optical elements 405 a, 405 b and 405 crespectively without interference from other optical elements, becausethe horizontal coordinates of the optical elements 405 a, 405 b and 405c are different from each other. In this example, the resolution of thelight engine 407 is high enough so that each optical element can beilluminated by a distinct light ray. Therefore, each optical element candisplay a color and brightness independent of other optical elements.

FIG. 5 is a diagram showing a possible layout of light engine pixels.The diagram further illustrates the relationship between the opticalelements and the light engine pixels in a volumetric display systemgenerally illustrated in FIG. 4. Each small square containing a numberrepresents a pixel. As shown, the 32×32 light engine pixels can bedivided into 8 x 8 tiles, each tile having 4×4 light engine pixels. The16 light engine pixels in any given tile can correspond to 16 opticalelements that lie on different layers, with the numbers associated witheach pixel (from 1 to 16) denoting the layer. For example, all the lightengine pixels marked “8” can correspond to optical elements that lie onlayer S. Therefore, a display block with 16 layers, each with 8×8elements can be constructed so that each element can be illuminated by adifferent light engine pixel without being interfered with by otherelements (assuming that light rays coming from the pixels are paralleland distinct). In other words, for that display block, there is aone-to-one correspondence between its optical elements and the lightengine pixels.

FIG. 6 is a flow chart illustrating the steps for displaying 3D imagesusing a volumetric display system generally illustrated in FIG. 4. Thefirst step 601 is to generate a 3D image to be displayed in a particulardisplay block. The 3D image is in the format of a 3D data array, whereineach element in the data array corresponds to a passive optical elementin the display block and contains color and brightness information ofthe optical element. The second step 603 is to transform the 3D imageinto a 2D image wherein the 2D image contains depth information, amongother information, of the 3D image. An example format the 2D image isillustrated in FIG. 5. The 2D image is a 2D data array, wherein eachelement in the data array corresponds to a light engine pixel andcontains the color and brightness information of the pixel.Transformation of the 3D image into the 2D image is done by mappingelements in the 3D data array to the 2D data array according to theone-to-one correspondence between the optical elements and the pixels.The 2D image contains depth information because pixels in the 2D imagecorrespond to different layers of optical elements. The third step 605is to project the 2D image onto the display block, resulting in a 3Dimage. Using the same steps, animated 3D images can be displayed byrepeatedly generating 3D images, transforming them into 2D images, andprojecting them in the display block.

FIG. 7 is a cross-sectional view of an example of a volumetric displaysystem. This example of a volumetric display system is of the type ofthe system generally illustrated in FIG. 4, in the sense that thepassive optical elements 703 in the system are regularly spaced withinthe display block 701. The light rays 705 are parallel to each other.Each light ray illuminates a different optical element. Because of thisregular spacing, there may be viewing directions (angles from which aviewer looks at the display) at which optical elements are occluded byother optical elements. Therefore, it may be advantageous to break theregular structure of this grid. This may be done for example by placingthe optical elements at locations that are random or semi-randomperturbations of their original locations in the grid as described ingreater detail in connection with FIGS. 8-10.

FIG. 8 illustrates an example of a configuration of optical elements 703in a grid in which they are slightly perturbed on the Z-axis in a randomfashion. Note that each light ray still illuminates a different opticalelement.

FIG. 9 illustrates an example of a configuration of optical elements 703in a grid in which they are perturbed on the X-axis in a semi-randomfashion. As shown, the optical elements 703 are no longer uniformlyspaced at each depth, but instead semi-randomly spaced so that there isa less regular structure while each light ray can still illuminate adifferent optical element, preserving image resolution. To ensure thateach light ray can still illuminate a different optical element, theoptical elements 703 are not perturbed with an arbitrary distance.Instead, they are perturbed with unit distances, each unit distancebeing the distance between the centers of two adjacent light rays. Also,the perturbation of one optical element will limit the locations whereother optical elements can be perturbed. As should be apparent to one ofordinary skill in the art, the perturbation of elements 703 illustratedin FIG. 9 could additionally or alternatively be performed in the Y-axis(not shown).

FIG. 10 illustrates an example of a configuration of optical elements703 in a grid in which they are slightly perturbed on the Z-axis in arandom fashion as with FIG. 8 and also perturbed on the X-axis in asemi-random fashion as with FIG. 9. Each light ray 705 still illuminatesa different optical element. As should be apparent to one of ordinaryskill in the art, the perturbation of elements 703 illustrated in FIG.10 could additionally or alternatively be performed in the Y-axis (notshown) in a semi-random fashion as with FIG. 9.

The perturbations of optical elements 703 illustrated by FIGS. 8-10break the regular structure of the grid of elements 703. This makes thestructure less distracting to the viewer. It also increases thevisibility of elements 703 as a function of the viewing angle of theviewer. In the case where optical elements 703 are perturbed in allthree dimensions, the number of elements visible to the viewer mayremain more or less constant when the viewer moves around the displayblock 701.

Although the perturbations in the Z-axis in FIGS. 8 and 10 are describedas being random, it should be apparent that the perturbation could bemade using any suitable technique, such as non-random or semi-randomplacement. As used in connection with FIGS. 8 through 10, semi-randomplacement refers to placement according to placement that is used toincrease the visibility of optical elements 703 but that is notaccording to a completely random technique.

In order to illuminate the optical elements 703, it is desirable to keeptrack of the locations of the optical elements 703 when manufacturingthe display block 701. Alternatively, suitable optical detectors couldbe used to determine the locations of the optical elements 703, forexample by sweeping the display block 701 with a point light source inone axis while observing reflections (and hence optical elements) in theother two axes. Keeping track of the locations of the optical elements703 in the display block 701 is desired because these locations are usedin the transformation of a 3D image to the 2D image to be projected. Forexample, when an optical element is perturbed in the X-axis asillustrated in FIG. 9, the optical element is no longer illuminated bythe same light ray (i.e., the light ray produced by the same lightengine pixel). Therefore, in order to produce (approximately) the same3D image, the light rays from the light engine needs to be rearranged.In other words, the 2D image to be projected needs to be changed.

FIG. 11 is a perspective view of another example of a volumetric displaysystem. Similar to the system generally illustrated in FIG. 4, passiveoptical elements 1107 are located on layers 1103 that are parallel toeach other. Light rays 1101 from the projector 1105, however, are notparallel to each other. It is possible to rearrange the location of theoptical elements 1107 so that each can be illuminated withoutinterference from other elements.

FIG. 12 illustrates a volumetric display system, according to variousembodiments of the present invention. In this system, a 2D image 1201 isprojected by a light engine 1207 to form a 3D image 1203 in the displayblock 1205 by simultaneously illuminating columns of passive opticalelements in the display block 1205. FIG. 13 is a cross-sectional view ofthe volumetric display system illustrated in FIG. 12. As shown, opticalelements 1301 form a grid inside the display block 1205, and eachvertical column of the grid is illuminated by a distinct light ray fromthe light engine 1207. Therefore, points in the 2D image 1201 aretransformed to columns of points in the display block 1205, resulting ina 3D shaped image 1203. As shown in FIG. 13, the optical elements 1301within each column may be perturbed in the Z-axis (vertical) directionto improve the appearance of the 3D image 1203 by increasing thevisibility of the elements 1301 and removing the appearance of a regularstructure of the elements 1301. It should be apparent to one of ordinaryskill in the art that this aspect of the invention could be implementedwithout perturbing the elements 1301. 3D images created in thisvolumetric display system may also be animated and put in motion ifdesired.

FIG. 14 is a cross-sectional view of another embodiment of thevolumetric display system, which can also be used to display 3D images.As illustrated, the passive optical elements of the top-level planeinside the display block 1205 are not aligned with other opticalelements in the Z-axis. Thus, each optical element on the top-levelplane may be illuminated by a distinct light ray from the light engine,without being obstructed by other optical elements. This enables thedisplay of 3D objects with a top surface, which can be as signedarbitrary textures. Textures of vertical side surfaces of 3D objectsdisplayed by this system can vary in the horizontal plane. The opticalelements within the columns can be perturbed in the vertical directionto increase the visibility of points. 3D images created in thisvolumetric display system can also be animated and put in motion ifdesired.

Interactive Volumetric Display Systems

In a volumetric display system according to various embodiments of thepresent invention, the 2D images being projected, whether static oranimated, can be generated using any suitable equipment. Furthermore, insome embodiments, a user can interactively control the generation of the2D images while viewing the projected 3D images.

FIG. 15 illustrates an example of such a system. As shown, a computer1503 generates and feeds 2D images to a video projector 1505, which thenprojects the images into the display block 1507 (e.g., a glass block) inorder to create 3D images. In some embodiments, a user of the system maycontrol the generation of the images in the computer using an inputdevice 1501, while simultaneously viewing the 3D images.

Some embodiments of the present invention may be used to implement agame. In this game, a volumetric display system is used to displayvarious 3D objects. Through a controller, a viewer controls the motionof one or more objects being displayed, and seeks to achieve goalspredetermined in the game design. For example, as illustrated in FIG.16, a 3D game similar to Pacman® may be implemented. As shown, a seriesof passageways 1601 may be created in three dimensions in a displayblock 1603 (e.g., a glass cube). The viewer controls the movement of agame character in these passageways 1601 and captures targets located atvarious locations in the passageways 1601 While the traditional Pacman®game is played in two dimensions, the new game further challenges theviewer to find the targets in a 3D maze. The display block 1603 may beilluminated through its bottom surface using an orthographic lightengine.

A projection of the passageways 1601 and locations in the glass cube1603 is illustrated in FIG. 17. As shown, the bold lines are passageways1601 in the maze with different layer levels as indicated. Thepassageways 1601 are formed by optical elements. The small squares 1701are locations along the passageways 1601. The viewer can move the gamecharacter 1703 along the passageways 1601 in three dimensions, which isachieved by continuously illuminating the positions of the gamecharacter 1703 in the passageways 1601 with a certain color. When thegame character 1703 moves to a small square, the square is illuminatedby a different color, notifying the viewer that the target has beencaptured.

Other Variations

FIG. 18 illustrates another embodiment of a volumetric display system.As shown, instead of a projector, a laser pointer 1801 is used as thelight source. In this example, the pointer rotates so that its beamraster scans the area of the Fresnel lens 1803. The scanning process isfast such that a scan of the whole area of the Fresnel lens 1803 iscompleted within the integration time of the human eye. During thescanning, the pointer 1801 is turned on and/or off for each etched gridpoint, based on whether the 3D image requires the point to glow or not.

In a volumetric display system, a display block containing opticalelements can be produced by means other than etching a glass or aplastic block using LID. FIGS. 19-25 illustrate various embodiments ofthe present invention wherein the display block is produced by stackinglayers of transparent material.

FIG. 19 illustrates optical elements 1901 made of a diffusely reflectingmaterial, positioned on a transparent layer 1903. In this example, thediffusely reflecting material can be the Spectralon paint produced byLabSphere Inc. This material closely approximates Lambertianreflectance. A Lambertian reflector scatters incident light in alldirections such that its radiance is the same from all viewingdirections that lie within the hemisphere from which the reflector isvisible. The radiance can be calculated as L=(ρ/π) n·s, where ρ is thealbedo or reflectance of the surface, n is the surface normal and s isthe source direction. As a result, the radiance of a Lambertian materialwill be independent of the viewing direction v. However, the areasubtended by each element from the viewpoint of the viewer varies as afunction of the viewing direction v. This is referred to asforeshortening, which causes the effective brightness of each element tofall off as n·v. This problem can be partially remedied using areflector that is convex in shape rather than planar. In someembodiments, optical elements are hemispherical Lambertian reflectors.In this case the brightness of the optical elements falls off moregradually.

FIG. 20 illustrates optical elements 2001 that are curved specularreflectors 2001, positioned on a transparent layer 2003. Thesehemispherical reflectors 2001 provide more uniform distribution ofbrightness with respect to viewing direction. In some embodiments, eachelement can be a steel ball of the type used in ball bearings. In thiscase, the incident light 2005 is reflected in all directions except avery small section in the lower hemisphere. Moreover, because eachreflected ray in any given direction is reflected by a single point onthe sphere, the brightness of the element is more or less uniform withrespect to the viewing direction.

FIG. 21 illustrates optical elements 2101 that are light diffusers 2101,located on a transparent layer 2103. In some embodiments, the opticalelements 2101 are created by roughening the surface of the transparentlayer 2103 within the area bounded by the elements. In some embodiments,the transparent layer 2103 is a plastic sheet, and the elements arecreated by etch scattering patterns using a plastic printer. Incidentlight 2105 can be received by the diffuser 2101 from either above orbelow the layer. As shown, the incident light 2105 strikes the diffuser2101 from below the layer and is scattered by the diffuser 2101 in alldirections within the upper hemisphere.

FIG. 22 illustrates a configuration of a volumetric display system usingreflective elements. In this system, the display block 2201 includesreflective elements placed on a stack of transparent layers. The viewer2203 is located at the same side of the display block 2201 as theprojector 2205. The viewer 2203 can move around the display block 2201and view different aspects of the scene (not shown) being displayed.When the viewer 2203 moves, each eye receives a new perspective of thescene (not shown). Also, multiple viewers can view the 3D scene (notshown) being displayed simultaneously as there is no conflict ofresources between the viewers.

FIG. 23 illustrates a configuration of a volumetric display system usingdiffusive elements. In this system, the display block 2301 includesdiffusive elements placed on a stack of transparent layers. Thediffusive elements are illuminated from the projector 2303 side and theimage is viewed from the opposite side. Again, multiple viewers may viewthe displayed 3D scene at the same time.

FIG. 24 illustrates another example of a compact volumetric displaysystem. In this system, the display block 2401 includes diffusiveelements placed on a stack of transparent layers. The light source 2403can be a liquid crystal display (LCD). Alternatively, it can be an arrayof densely packed light-emitting diodes (LEDs), or any other suitablelight source. A lenslet array 2405 made of very small convex lenses canbe used to convert the light from each pixel on the LCD or LED array2403 into a parallel beam, focused on a single diffusive or reflectiveelement. A backlight 2407 can be placed near the LCD array. Therefore,the display can be one self-contained unit.

Another alternative to producing a display block is to mold Lambertianreflectors, specular reflectors, or diffusers into a single plasticcube. The plastic solid can be coated with anti-reflection coating toensure that reflections from its outer surface are minimized.

FIG. 25 illustrates another volumetric display system using multiplelight sources 2501. This remedies the situation where light rays from asingle projector do not have full access to all the elements in adisplay. In this example, if one projector does not have access to aparticular display element, due to obstruction by another element,another projector can be used to access the element. Multiple projectorscan also be used to enhance the quality of the displayed image. When allprojectors have access to all the display elements, each element can becontrolled with a greater brightness range than when using a singleprojector.

Animating Images on a 3D Surface

According to various embodiments of the present invention, passiveoptical elements of the volumetric display system can be positioned toapproximate a predetermined 3D surface, instead of forming a 3D grid.

Passive optical elements approximating a 3D surface can be made close toeach other subject to the structural limitations of the display block.As long as a 3D surface of elements within a certain region is notparallel to, or close to being parallel to, the direction of theilluminating light rays, elements close to each other on a 3D surfacecan be illuminated by distinct light rays. In this way, the 3D surfacein that region may be illuminated with a high resolution image.

FIG. 26 illustrates such a volumetric display system. In this system,the optical elements inside a display block 2601 form a 3D surface 2603that models a human face. A 2D image (not shown) is projected onto this3D surface 2603. For example, the 3D surface 2603 may have a similar 3Dshape to that of a costume mask that only covers the front part of awearer's face. The 2D color face image (not shown) and the 3D surface2603 formed by the optical elements can both be created from the samehuman face, if desired, to give the most realistic appearance.Similarly, the more closely the features and proportions of the facethat is projected onto the 3D surface 2603 corresponds to that surface,the more realistic the 3D face image will appear.

To create a 3D face surface 2603, a 3D contour of a face of a model maybe captured using any suitable technique. Using the captured 3D contour,the locations of the optical elements of the 3D surface can bedetermined by applying known methods, such as a meshing algorithm. Theface may then be created (e.g., etched) in a block of glass 2601 using,for example, a LID technique.

If a 2D animated video showing facial movements is projected on the 3Dsurface 2603, the 3D surface image may appear to be animated. Althoughthe 3D surface 2603 representing the face is static, the end result canbe a 3D surface image that appears to show facial movements in arealistic manner when the deformations of the face during the facialmovements are small. In certain embodiments, it may be desirable toproject a talking face on the 3D surface 2603 while playing acorresponding spoken voice to give a life-like appearance.

Because the projection of video is likely to be done from a viewpointthat differs from the one used to capture the video, a transformationmay be needed to morph the captured video before it is projected ontothe 3D face surface. To enable this transformation to be achieved, itmay be necessary to select a set of points (e.g., 100 points) from oneframe in the captured video, and a corresponding set of points on the 3Dface surface. For example, the user may select “inside corner of lefteye” on the frame by moving a cursor on the display showing the frameand pressing a button, and on the face surface by moving a dot projectedonto the face surface and pressing a button. In this way, the selectedpoints on the 3D surface can be located at the positions where thepoints in the captured video should be projected i.e., the image andsurface are registered (or put in registration).

This set of points on the surface and image may then be used to create amesh of triangles on the surface and the image. For example, DelauneyTriangulation, as known in the art, may be used to create a mesh oftriangles on each. Next, any suitable texture mapping technique, such asOpen GL, may be used to take the texture from each triangle of the imageand map it to the corresponding triangle of the surface. The texture ofthese surface triangles may then be projected onto the surface todisplay the image. In this way, the image is morphed onto the surface.

Alternatively, frames in the captured video can be morphed using otherknown methods, such as a polynomial or spline model, to compute theto-be-projected video. When the computed video is projected, the facialfeatures show up at desired locations.

The process of animating a 3D face image using a static 3D surface of aface 2603 is illustrated by a flow chart in FIG. 27. As shown, the firststep 2701 is to capture the 3D contour of a face. The second step 2703is to create a 3D surface formed by optical elements inside a displayblock. The 3D surface approximates the face captured. The third step2705 is to capture a 2D video containing the facial movements of theface. The fourth step 2707 is to register a frame in the 2D video withthe 3D surface in the display block. This can be achieved by selecting anumber of points from the frame and register the points withcorresponding points on the 3D surface. The fifth step 2709 is to mapthe frame in the 2D video onto the 3D surface. The next step 2711 is toproject the mapped 2D video frame on the 3D surface. After that, steps2709 and 2700 are repeated for another frame in the 2D video, so that ananimated 3D image is created.

According to various embodiments, an interactive volumetric displaysystem that animates a talking face can be controlled interactively byhuman speech. For example, the system can further include face and/orrecognition capabilities. The system may use face and/or voicerecognition to detect the user and spoken questions and/or commands ofthe user. In response to questions and/or commands, the system maydisplay face images on the 3D face surface that respond to the user. Forexample, these images may include animated video clips and/or beaccompanied by audio. In some embodiments, certain video clips may beseamlessly stringed together to create a long video presentation bylinking clips with similar ending and starting frames. In this way, thesystem can create the impression that the face inside the glass block isanswering questions asked, and can do so dynamically.

FIG. 28 is a cross-sectional view of another example of a volumetricdisplay system. In this system, the 3D surface 2801 formed by theoptical elements is a sphere. As shown, the optical elements making upthis surface 2801 are arranged so that some light rays 2803 strike theelements on the side closest to the light engine 2805 while other rays2803 pass through this side and project onto the distant side. In orderto project and animate images on the entire 3D surface 2801 using asingle light engine 2805, each optical element on the surface 2801 ispreferably positioned so that a distinct light ray will be able to reachand illuminate it.

A 2D image to be projected onto the 3D surface 2801 may be formed fromtwo interlaced 2D images. One of the two interlaced 2D images is to beprojected onto the hemisphere closer to the light engine 2805, and theother 2D image is to be projected onto the other hemisphere. These two2D images may be projected at the same time onto the 3D surface 2801, ormay be rapidly displayed in alternating frames (or another suitablesequence of frames). By projecting moving video onto the opticalelements, the surface 2801 could appear to be moving. For example, the3D image could be configured to imitate a spinning sphere.

FIG. 29 illustrates an example of a technique using which one image maybe projected on one surface of a sphere 2801 while another image isprojected on the opposite surface of the sphere 2801. As shown in alarge-scale form for ease of illustration, black areas 2901 within thecircle represent light rays that are directed at optical elements on onesurface whereas white areas 2903 within the circle represent light raysthat are directed at optical elements on the other surface.

Correspondence of Image Pixels to Optical Elements

Because the number of pixels in a 3D image may have a higher resolutionthan the number of optical elements on which the image is to beprojected, it may be necessary to transform the image to a lowerresolution using any suitable technique. For example, in accordance withcertain embodiments of the invention, the following algorithm may beused. For each optical element, an ellipsoid may be calculated with theoptical element at its center. In some embodiments, this ellipsoid maybe a sphere. Next, it is determined which pixels in the image fallwithin the area of the ellipsoid, and any such pixels are placed in alist for that ellipsoid. Preferably this list is generated for eachoptical element in advance, though it may also be generated when theimage is being generated if acceptable performance is achieved. Next,for each list, a weighted average of the color components of each imagepixel in this list is calculated. The weighting may be based on, forexample, each pixel's distance from the point in the image correspondingto the optical element so that the pixels furthest from that point aregiven the lowest weighting. These weighted averages are then used to setthe color of each optical element. As will be apparent to one ofordinary skill in the art, any suitably sized ellipsoids may be used,and the ellipsoids for neighboring elements may overlap. Preferably, theellipsoids are equally sized as small as possible while still includingevery pixel in the image.

Increased Brightness of Volumetric Display

If increased brightness at a certain point in a volumetric display isdesired, it can be achieved by etching multiple optical elements closeto said point using any suitable technique, and illuminating theresulting cluster of optical elements. For example, in one approach,four elements may be positioned in a plane in the shape of a square andilluminated, for a total of four elements. In another approach, fourelements may be positioned in a plane in the shape of a square and twomore elements located equidistant from the center of the square on anaxis perpendicular to the plane, for a total of six elements. In yetanother approach, two squares of four elements may be located inparallel planes with the one square rotated 45 degrees with respect tothe other, and two more elements each positioned on an opposite side ofthe two squares along an axis that passes through the centers of the twosquares and that is parallel to the two planes, for a total of tenelements.

3D Display Systems Using Anamorphic Light Engines

FIGS. 30-39 illustrate systems for creating a high resolution 3D imageon a 2D screen using anamorphic light engines. An anamorphic lightengine, as used in this context, is a projector system that compressesimages while projecting them.

FIGS. 30 and 31 illustrate the steps involved for capturing anddisplaying a 3D scene 3001 in an example of a 3D display system usinganamorphic light engines. In the first step 3101, an array of k videocameras 3005 is used to capture the 3D scene of interest 3001. At somepoint prior to this step, the cameras are preferably put in registrationwith each other, or a suitable algorithm is used to put the images inregistration with respect to each other. At each instant of time, thecamera array 3005 produces a set of k images 3007. In the second step3103, these images 3007 are processed to obtain a set of in projectionimages 3009. For instance, the first k columns in the first projectionimage are obtained by concatenating the first columns of all the cameraimages 3007 in the same order in which they are arranged in camera array3005. The next set of k columns in the first projection image isobtained by concatenating the second columns of all the camera images3007 in the same order that the first columns were concatenated. Theprocess is repeated for all columns needed to make up the firstprojection image. This process is then repeated for subsequentprojection images by starting with the columns in the camera images 3007following the last column used to form the previous projection image. Inthe third step 3105, these images 3009 are fed into the array of inprojectors 3003. The projectors 3003 are coupled with anamorphic lenses3011, which compresses the projection images 3009 in the horizontalaxis. The compressed images are displayed on a diffusive screen 3013,and the compressed images on the screen 3013 are viewed through alenticular lens array 3015. In this way, as a user views the display,the user will see particular columns of the camera images depending onthe angle at which the user is viewing the display. In particular, oneeye of the user would view images at one angle while the other eye wouldview images at a different angle, thereby allowing the user to perceive3D scenes.

FIG. 32 illustrates a single anamorphic light engine in the systemillustrated in FIG. 30. As shown, the anamorphic light engine includes adigital projector 3201 and an anamorphic lens attachment 3203. The lensattachment 3203 is used in conjunction with the projector 3201 tocompress the image produced by the projector in the horizontaldirection. This lens attachment 3203 in its simplest form can be acylindrical lens that compresses the image produced by the projectorlens in one of its two dimensions. Additionally, the anamorphic lensattachment 3203 can include corrective lenses to compensate for thechromatic aberrations, astigmatism, and other effects produced by thecylindrical lens.

FIG. 33 illustrates the compression of an image by the anamorphic lightengine illustrated in FIG. 32. As shown, the input image 3301 iscompressed to the output image 3303 by a factor of k. The width of theimage is reduced by a factor of k while the brightness of the image isincreased by a factor of k.

FIG. 34 illustrates another example of an anamorphic light engine. Theanamorphic light engine in this example uses a curved mirror 3401instead of an anamorphic lens. Given any mapping from the projectorinput image to points on the screen, one can design a mirror that, inconjunction with the lens of the projector, can achieve (or closelyapproximate) the mapping. The shape of the mirror may be computed bysolving a numerical optimization problem that minimizes distortions inthe final screen image.

FIG. 35 illustrates a linear array of anamorphic light engines 3501 inoperation. The anamorphic light engines 3501 are generally illustratedin FIG. 32. As shown, a group of m projectors 3503 are stacked closelytogether such that the compressed images 3507 they produce are tiledtogether without gaps to create a single image. The image produced hasthe same size as a normal image but the horizontal resolution isincreased by a factor of k. For example, if the projectors have aresolution of 1024×768 pixels and produce a normal image of size 24inches ×18 inches, and the anamorphic lenses 3505 have a horizontalcompression factor of 24, the image produced by the projectors wouldstill have 1024×768 pixels but would be 1 inch ×18 inches in size. 24such projectors 3503 can therefore be stacked together to create a 24inches ×18 inches image with 24576×768 pixels. That is, each pixel wouldbe 0.023 inches (595 microns) in height and 0.00097 inches (24.80microns) in width.

Using the anamorphic light engine array 3501 illustrated in FIG. 35, 3Dimages can be displayed according to various embodiments of the presentinvention. FIG. 36 shows a diffusive screen 3601 and a lenticular lensarray 3603, on which a tiled image 3605 is projected. Diffusive screen3601 removes the directional property of the rays projected by theprojectors (not shown). That is, when viewed from the side opposite fromthe projectors, the same high resolution image 3605 is seen but eachpixel appears more or less equally bright from all viewing direction.The lenticular lens array 3603 is placed in front of the diffusivescreen 3601 at such a position that the lenticular lenses 3603 arealigned with the vertical direction of the image 3605. Each lenticularlens is associated with a large number of image lines. For example, alenticular lens array 3603 with 40 LPI (lenticules per inch) can be usedwhere each lenticular lens has a viewing (divergence) angle of 30degrees (+15 to −15 degrees). If 24 projectors (not shown), each with1024×768 pixels are used to create a 24 inches ×18 inches tiled image3605, each lenticular lens on the array 3603 will include 1024/40 =26.05image lines. Because each lens has a horizontal field of view of 30degrees, this would give us an angular resolution of approximately 1.15degrees. Therefore, the images received by the left and right eyes of aviewer can be different (because at normal viewing distances, the eyeswould be separate by more than 1.15 degrees), allowing the viewer toperceive the displayed scene in three dimensions. The viewer is alsofree to move around within the region bounded by the viewing angle ofthe lenticular lenses 3603.

FIG. 37 illustrates an alternative to the embodiment illustrated in FIG.36. In this example, a parallax barrier array 3701, instead of alenticular lens array 3603, is placed in front of the diffusive screen3601. The effect created by a parallax barrier array 3701 is similar tothe effect created by a lenticular lens array 3603. For example, aparallax barrier array 3701 with 40 vertical apertures (slits) per inchcan be used where each parallax barrier has a viewing (divergence) angleof 30 degrees (+15 to -15 degrees). If 24 projectors (not shown), eachwith 1024×768 pixels are used to create a 24 inches ×18 inches tiledimage 3605, each parallax barrier on the array 3701 will include 1024/40=26.05 image lines. Because each parallax barrier has a horizontal fieldof view of 30 degrees, this would give us an angular resolution ofapproximately 1.15 degrees.

FIG. 38 illustrates another embodiment of a 3D display system usinganamorphic light engines. As shown, this embodiment is a compact rearprojection display system. The system includes an array of compactanamorphic light engines 3801, each producing a 2D image. The lightengines 3801 are compact in the sense that the system uses compactanamorphic lenses (not shown) to directly produce the compressed image3807, rather than using an optical attachment to convert the perspectiveoptics of a projector into an anamorphic one. To make the system morecompact, a planar mirror 3803 may be used to fold the optical pathbetween the light engines 3801 and the screen 3805 as shown.

The light engines 3801 used in the system illustrated in FIG. 38 can bemade from off-the-shelf components, as illustrated in FIG. 39. As shown,this engine can include a light source 3901, a digital micromirrordevice (DMD) 3903, a lens 3905 that collimates the light from the lightsource 3901 and focuses it onto the DMD 3903, and an anamorphic lens3907. The DMD 3903 can be from Texas Instruments. The DMD 3903 modulatesthe incident light based on the input image and reflects the light. Thereflected light can be projected onto the projection screen 3805 (shownin FIG. 38) using a custom-made anamorphic lens 3907. In someembodiments, a liquid crystal on silicon (LCOS) device can be usedinstead of the DMD. Again, in some embodiments, the light source 3901can be a medium-powered LED. Alternatively, the light source 3901 can bea high-powered Xenon lamp of the type used in projectors.

Embodiments of systems generally illustrated in FIGS. 30-39 may also beused as high resolution projection systems for projecting 2D images. Theenhanced resolution in one of the two dimensions of an image can be usedto dramatically improve the perceived resolution of an image.

3D Display Systems Using Light Field Elements

According to various embodiments of the present invention, 3D displaysystems can use 2D light field elements to measure and create afour-dimensional (4D) light field. A light field is defined as afunction that describes the amount of light traveling through everypoint in a region in every possible direction. A 2D light field, as usedin this context, is a light field wherein the region only includes asingle point. (This light field is referred to as a 2D light fieldbecause every light ray passing through a single point has twodimensions, in the sense that the ray can be represented by two angleswith respect to the point.) A 4D light field, as used in this context,is a light field wherein the region is a 2D plane. (Again, this lightfield is referred to as a 4D light field because light rays passingthrough a single point have two dimensions, and the plane has twodimensions.) A light field element is defined in this context as adevice that either measures or produces a 2D light field from a singlepoint in space. A bi-directional light field element is defined in thiscontext as a light field element that can simultaneously measure andproduce a 2D light field from the point where the light field element islocated. Light field elements can be arranged in a 2D array to measureand/or produce 4D light fields.

Because a 4D light field contains directional information of light raysat each point in a 2D plane, it can provide different images indifferent directions. As a result, the two eyes of a viewer can receivedifferent 2D images from a 4D light field. Therefore, a 4D light fieldcan be used to display a 3D scene for viewers around the light field.

FIG. 40 illustrates a 3D display system using light field elements. Asshown, the system includes a 2D array of reflective spheres 4001, whichcan be curved reflectors (e.g., curved mirrors). In some embodiments,other shapes, such as paraboloidal, hyperboloidal, conical andellipsoidal can be used instead of a sphere. A digital projector 4003 isused to project light rays onto the array of spheres 4001. A camera 4005is used to capture the light rays reflected by the spheres 4001. Alsoshown, a half-mirror 4007 is located between the projector 4003 and thecamera 4005. The projector 4003 and the camera 4005 are located inpositions such that the projection centers of the projector 4003 and thecamera 4005 are coincidental. Therefore, the rays that are used toproject and the rays that are used to measure the 4D light fieldcoincide.

The combination of the projector 4003, camera 4005, and each sphere is asingle bi-directional light field element. This is because each spherecan reflect light rays from all directions in the space above thespheres 4001 to the camera 4005, so that almost the entire space abovethe spheres 4001 is captured (a small part of this space will beobstructed by neighboring spheres). Similarly, the projector image thatis projected onto each sphere will determine the distribution of lightreflected by the sphere. Therefore, each sphere is used to measure andproduce a 2D light field, where the two dimensions are the angles ofrays with respect to the center of the sphere.

Because a 2D array of spheres 4001 is included, the system is able tomeasure and produce a 4D light field. For example, the projector 4003and the camera 4005 can have 1000×1000 pixels each and their fields ofviews can be focused on an array of 50×50 spheres 4001. Each spheretherefore corresponds to a 20×20 image in the camera 4005 and theprojector 4003. More precisely, each sphere occupies a circular imagehaving diameter of 20 pixels. This image of a single sphere captured bythe camera 4005 can be used to determine the light incident upon thesphere from all directions with respect to the center of the sphere. The4D light field is measured so that the system can be used to display a3D scene together with the effect of the lighting environment upon the3D scene. In some embodiments, a camera is not included, so the lightfield elements can only produce a 4D light field.

FIG. 41 illustrates the operation of the system generally illustrated inFIG. 40. In this example, a 3D scene (in the shape of a cube) 4101 isbeing displayed by the system. In some embodiments, the 3D scene 4101 tobe displayed can be computer-generated. Alternatively, the 3D scene canbe based on captured images. Two light sources 4103 and a viewer 4105are present in the environment. A point on the surface of the cube 4107is shown. To show the effect of the light sources 4103 on the cube 4101,the system measures two light rays 4109 that would have (approximately)shone on the point 4107. These two light rays 4109 are reflected by twospheres 4117 into a camera 41 11. Therefore, the locations andbrightness of the light sources 4103 as seen by the point 4107 aremeasured. The brightness, colors and directions of the two rays 4109 canbe used to compute the brightness and color of the point 4107 in thedirections of the two eyes of the viewer 4105. This computation may takeinto account the reflectance properties of the cube's surface. As shownin the figure, the two rays 4113 that would be directed from the point4107 towards the viewer's 4105 eyes can be produced by the projector,which is at the same location as camera 411 1, by illuminating twopoints on another sphere 4115. This process of measuring light rays,computing, and projecting onto the sphere may be repeated (or performedsimultaneously) for each point onto which the projector can projectlight. This process may also be performed continuously to account forchanges in the light in the environment. The camera and the projectorshould be put in registration with respect to each other prior toperforming this process using any suitable technique. In someembodiments, the camera may only need to observe a few or one of thespheres to measure the impact of the environment rather than observingall of the spheres. Also, in certain embodiments, the camera may beomitted and the image that is generated by the projector is createdindependently of the environment.

FIG. 42 can be used to illustrate more generally the technique describedin FIG. 41. As shown, two parallel planes 4201, 4203 are placed oneither side of the array of spheres 4001. A complete 4D light field canbe represented using points on these two planes 4201, 4203. For example,if we denote points on the upper plane 4201 as (u, v) and points on thelower plane 4203 as (s, t), the incident 4D light field (which is theillumination of the environment) can be written as I(u,v,s,t) and thedisplayed light field (which is the 3D image being displayed) can bewritten as D(u,v,s,t). This is because for each point (u,v) on the upperplane 4201, every point (s, t) on the lower plane 4203 corresponds to adifferent direction of a ray passing through (u,v). The camera image ofthe spheres 4001 in camera 4111 gives us discrete samples of I(u, v, s,t). Each sample is obtained from a single sphere. These samples can beinterpolated using standard light field interpolation methods andre-sampled to obtain a dense representation of the measured field I(u,v, s, t). Given an incident 4D light field, we can use the geometric andreflectance properties of a 3D scene 4101 to compute the desired displayfield D(u, v, s, t). This computation of the display field D(u, v, s, t)for any given incident field I(u, v, s, t) and any given scene is doneusing standard computer graphics rendering methods. The best discretesamples that would approximate the computed display field can beprojected by a projector (not shown) onto the array of spheres 4001.

FIG. 43 illustrates another embodiment of the present invention. In thisembodiment, the light field elements include an array of spheres 4301arranged on three faces of a cube. The entire array of spheres 4301 ismeasured and illuminated using a single projector-camera system 4303.Alternatively, multiple projector-camera systems can be used to increasethe resolution of the measured and displayed light fields. The advantageof this system is that the viewers 4305, 4307 are free to move aroundthe array of spheres 4301 and observe 3D scenes (not shown) from alarger range of viewing directions.

FIG. 44 illustrates a bi-directional light field element that iscompact. As shown, the light field element includes a wide angle lens4401, an image detector 4403, a LCD display 4405 and a half-mirror 4407.The wide-angle lens 4401 creates images on the image detector 4403 andprojects images from the LCD display 4405 at the same time. In thisexample, the half-mirror 4407 is used behind the lens 4401 so that thelens 4401 can be shared by the detector 4403 and the display 4405.

FIG. 45 illustrates a 3D display system using a large array ofbi-directional light field elements 4505 illustrated in FIG. 44. Thelight field elements 4505 are located on a plane 4503. This system canmeasure the 4D light field falling on it as well as display a 4D lightfield. The displayed light field enables multiple viewers to experiencea displayed 3D scene 4501. In addition, the measured light field can beused to accordingly change the appearance of the displayed scene 4501.

FIG. 46 illustrates a 3D display system using unidirectional light fieldelements. Each element at any given location can only measure or displaya 2D light field. Elements that measure a 2D light field (one of whichis shown as 4603) and elements that produce a 2D light field (one ofwhich is shown as 4605) may alternately be located in a plane 4601. Thissystem can be used on large display systems (e.g., electronicbillboards), in which case resolution is less of an issue. Although thespatial sampling of both the measured and displayed fields is morecoarse in this case, it is sufficient for viewing from a large distance.

FIG. 47 illustrates another system using light field elements. In thissystem, a single LCD panel 4701 is used to generate the displayed lightfield. An array of image detectors 4707 is used to measure the incidentfield. A backlight 4705 is used to illuminate the LCD panel 4701. Thedetectors 4707 are placed between the backlight 4705 and the LCD panel4701. As shown in the figure, the detectors 4707 are placed in acheckerboard fashion such that the spaces not occupied by the detectors4707 are used to produce displayed light fields. In the areas occupiedby the detectors 4707, the LCD is inactive or simply used as anattenuator for the detectors 4707, which can be useful when the lightingin the environment is too bright for the detectors 4707 to measurewithin their dynamic range. In front of the LCD panel 4701 is a lensletarray 4703 (i.e., an array of small lenses). A lens in the array 4703that lies in front of a detector is used to form the detector image. Alens in the array 4703 that does not lie in front of a detector is usedto project the light field displayed on the LCD elements just behind thelens. The lenses of the array 4703 have a reasonably large field ofview, so that the system can measure illumination incident from a widerange of directions, and the viewers can view the displayed scene from alarge range of viewing angles.

FIG. 48 is a flow chart illustrating the steps for displaying 3D scenestogether with the effect of the environment on the scenes, using asystem illustrated in FIG. 40. As shown, the first step 4801 is togenerate a 3D scene using any suitable methods. The second step 4803 isto measure the 4D light field of the environment using the light fieldelements. The third step 4805 is to compute the 4D light field to bedisplayed by combining the generated 3D scene and the measured 4D lightfield of the environment. The last step 4807 is to display the 4D lightfield using the light field elements. This process may loop back to step4801 to account for changing scenes or step 4803 to account for changingenvironments.

Other embodiments, extensions, and modifications of the ideas presentedabove are comprehended and within the reach of one skilled in the artupon reviewing the present disclosure. Accordingly, the scope of thepresent invention in its various aspects should not be limited by theexamples and embodiments presented above. The individual aspects of thepresent invention, and the entirety of the invention should be regardedso as to allow for modifications and future developments within thescope of the present disclosure. The present invention is limited onlyby the claims that follow.

1. A system for displaying three-dimensional (3D) images, the systemcomprising: a plurality of projectors that includes a first set ofoptical elements that are configured to compress images in onedirection; a diffusive screen onto which compressed images from theplurality of projectors are projected; and a second set of opticalelements disposed near the diffusive screen that createsthree-dimensional images from the compressed images when perceived by aviewer.
 2. The system of claim 1, wherein the one direction is ahorizontal direction.
 3. The system of claim 2, wherein each projectorcreates horizontally compressed images that are tiled together withother light engines to form a composite image.
 4. The system of claim 1,wherein the first set of optical elements is one or more anamorphiclenses.
 5. The system of claim 1, wherein the second set of opticalelements is an array of lenticular lenses.
 6. The system of claim 1,wherein the second set of optical elements is a parallax barrier array.7. The system of claim 1, wherein the images that are compressed areformed from portions of separate images that are combined together. 8.The system of claim 7, wherein the images are formed by concatenatingvertical bands of the separate images together.
 9. The system of claim8, wherein each vertical band is a column of pixels.
 10. A system fordisplaying three-dimensional (3D) images, the system comprising: aplurality of light engines that includes a first set of optical elementsthat are configured to compress images in one direction; a diffusivescreen onto which compressed images from the plurality of light enginesare projected; and a second set of optical elements disposed near thediffusive screen that creates three-dimensional images from thecompressed images when perceived by a viewer.
 11. The system of claim10, wherein the one direction is a horizontal direction.
 12. The systemof claim 11, wherein each light engine creates horizontally compressedimages that are tiled together with other light engines to form acomposite image.
 13. The system of claim 10, wherein the first set ofoptical elements is one or more anamorphic lenses.
 14. The system ofclaim 10, wherein the first set of optical elements includes a planarmirror.
 15. The system of claim 10, wherein the first set of opticalelements includes a digital micromirror device.
 16. The system of claim10, wherein the second set of optical elements is an array of lenticularlenses.
 17. The system of claim 10, wherein the second set of opticalelements is a parallax barrier array.
 18. The system of claim 10,wherein the images that are compressed are formed from portions ofseparate images that are combined together.
 19. The system of claim 18,wherein the images are formed by concatenating vertical bands of theseparate images together.
 20. The system of claim 19, wherein eachvertical band is a column of pixels.
 21. The system of claim 10, whereinthe plurality of light engines further comprise a plurality of lightdetectors, a plurality of light sources that are interspersed throughthe plurality of light detectors, a plurality of lenses positioned suchthat light from the plurality of light sources is projected in multipledirections and such that light from multiple directions is directed tothe plurality of light detectors, and a plurality of liquid crystalspositioned between the plurality of lenses and the plurality of lightdetectors and light sources.
 22. The system of claim 21, furthercomprising a processor coupled to the plurality of light detectors andthe plurality of light sources, wherein the processor is configured toproject the light from the plurality of light sources based upon thelight received by the plurality of light detectors.